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ABSTRACT 

The National Assessment of Educational Progress is 
encouraging the interpretation of its data to ma'ke them more useful 
for local educators, and is facilitating the adaptation of NAEP 
procedures to state and local assessment programs. The Department of 
Utilization/Applications was formed in October 1971 to facilitate the 
use of technology developed and data produced by the commission. The 
implications of the NAEP model and data for local education are 
discussed. (Author) 
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Many financial and human resources have been invested in the 
National Assessment of Educational Progress (NAEP) since the mid-60 's. 
These resources have already given educators and the lay public some 
products. A sophisticated objectives-referenced assessment model 
which yields data about the educ^^cional attainments of national 
population groups was developed; concensus objectives in ten diverse 
subject areas have been produced and published; a full complement 
of objectives-referenced exercises have been developed for the ten . 
subject areas; by summer, data vriLll have been collected from a 
national sample for eight subject areas; and reports will soon be 
available for seven subject areas. Under the sponsorship of the 
Education Commission of the States (ECS) and with continuous 
funding by the United States Office of Education, National Assessment 
has overcome early criticism and solved the basic operational and 
technical problems involved in producing large amounts of objectives- 
referenced data. 

In terms of ' its original goals, National Assessment is well on 
its way to success. It has proven that it can provide comprehensive 
data about the educational attainments of young Americans, and it 
is presently collecting the first data which will be analyzed to 
measure the growth or decline of those educational attainments over 
time. Given only these goals, project staff could certainly 
continue to provide the data while refining and improving all aspects 



of exercise development, data collection, analysis and reporting. This 
type of future would correctly reflect most literature and thought 
about National Assessment prior to 1972. The model's reporting 
component was designed to present comparative percentages of ' 
success for national popula,tion groups and to disseminate reports 
widely. The final component, utilization of National Assessment to 
improve American education, was left to others. 

The logic for this was sound. The National Assessment staff is 
a technical, operational staff whose job is to provide data of high 
quality, not a group of educational, policy makers whose job is to 
interpret educational data and implement educational change. 
Proponents of the project believed educators and lay people, once 
they. knew the assessment data was available, would read it, interpret 
it, discuss it and use it to implement educational change. Also, 
those who believed strongly in the need for a national data base 
on the level of educational attainments and the change in that level 
over time were patient people. The long run return of' a more 
intelligent allocation of educational dollars would be, in and of 
itself, worth the investment of human and financial resources in the 
project. 

Some were not so patient, or they desired a more immediate 
return on the dollars spent each year. Others saw potential, short 
run uses of the data and model which were not being tapped by 
implementing a model which provided no systematic attempt at 
interpretation or utilization by NAEP, ECS or USOE. 



Overall, the data was accepted as high quality data vd.th a 

potential for use. However, there was little enthusiasm since that 

potential vas vague and futuristic. Two major reactions came from 

educators and laymen who looked at the reports. One was, "So what? 

You have given us this great amount of descriptive technical data, 

but nobody has attempted to tell us what it means for American 
education." The second was, "Can we get this kind of data at our 

educational autonomy level? We need objectivfes-^ref erenced data 

about population groups which are relevant to our decision making 

process. " 

National Assessment's Policy Committee responded to these 
reactions by recommending t:o the ECS Steering Committee, the creation 
of a Department of Utilization and Applications. This recommendation 
was approved in December, 1971. The major goals of National Assessment 
did not change, but the project altered the original model by 
adding an active utilization component to stimulate immediate use 
of the data and model. 

The Department of Utilization and Applications was organized 
into two distinct areas, each of which responds to one of the above 
reactions. The major goal of Data Utilization and Applications is 
to encourage the use of National Assessment materials and data by 
educators and laymen involved in all levels of educational decision 
making. This area of the department is attempting to generate 
dialogue which will answer the qi. tion, "So what?" and which will 
lead to changes in American education based upon that answer. The 
second area of the department is Model Utilization and Applications, 



and Its major goal is to facilitate the adaptation of the National 
Assessment process, technology and materials to state and local 
assessment programs. This area of the department is helping states 
and local districts borrow from National Assessment experience and 
products to produce objectives-referenced data which is relevant 
to decision making. 

Those who reacted to National Assessment .strongly enough to 

bring about the creation of the Department of Utilization and 
Applications will watch the progress of that department carefully. 

Many people will judge National Assessment by its direct, short run 
utility at state and local educational autonomy levels in addition 
to, or even instead of, its long run production of a comprehensive 
national data bank. It is certainly too early to make a final 
judgment about the project's utility at these levels. National 
Assessment is still in the first cycle of a pioneering effort; 
therefore, all procedures and products will be refined based upon 
initial experiences. The Department of Utilization and Applications 
is just over a year old; therefore, departmental goals and plans 
are still being defined, and only a few programs have been iniated. 
However, it is not too early to look at existing plans and programs 
to gain some insight into the possible implications of the National 
Assessment model and data upon state and local education. 

Data Utilizations and Applications has not taken on the role 
of interpreting NAEP data or suggesting implications for education. 
It has instead taken on the role of systematically generating, or 
catalyzing, in-depth interpretations of the data and statements 



concerning the educational implications of the data. Two major 
interpretative efforts have been funded to date. The National Science 
Teachers Association (NSTA) presently has a team doing a comprehensive 
in-depth study of National Assessment's science reports to answer the 
question, "NAEP findings in science: What do they mean?'' The team, 
which includes scientists and science educators, will thoroughly examine 
the science data, interpret it, discuss its educational implications 
for education and teaching and prepare appropriate reports. The NSTA 
will publish and disseminate the reports and utilize the findings of 
the study in national and regional conventions. The results are 
due this spring. The National Council for the Social Studies (NCSS) 
has just begun a study which will be completed this fall. Study 
teams will thoroughly examine all National Assessment procedures and 
measures. Conclusions about procedures and measures will then be 
taken into account by study teams interpreting the project's 
citizenship and social studies data. Also, another study team will 
investigate the potential utility of parts of the NAEP model in local 
schools to improve curriculum and instruction. The NCSS will publish 
reports about all phases of the Rtudy, including the educational 
implications of the data and model. These findings will also be used 
in national and regional conventions. 

Data Utilization and Applications presently plans to generate 
an in-depth interpretative effort in each subject area. Some will 
involve contracts with educational organizations such as the NSTA and 
NCSS, but others will involve special task forces composed of scholars, 
educators and laymen who will bring more diverse viewpoints to the 



task of interpreting National Assessment data and discussing its 
educational implications. These efforts, along with National 
Assessment staff's redesign of basic reports to make them more 
interpretable, will present educators and the lay public with reports 
from which conclusions about educational implications can be more 
readily dram. These reports will present some very evident 
educational implications of the data and will stimulate further 
dialogue around more controversial ones. When educational decision 
makers use these findings based upon NAEP data to support points of 
view about- recommended curiiculum and instructional changes. National 
Assessment will have an impact upon American education at whatever 
autonomy level this occurs. 

In October, 1972, Data Utilization and Applications also 
discussed every aspect of the project in a two-day session with 
members of the Council of Great City Schools. As a result of this 
meeting the Council's Board of Directors passed a resolution 
communicating its deep concern about National Assessment's methods 
of reporting and disseminating data relating to urban areas. The 
resolution recommended that National Assessment and the Council of 
Great City Schools cooperate in the dissemination of the results and 
attempt to establish more representative sampling of urban areas. 
Cooperation and investigation have begun in a preliminary mannoir^ The 
Council further recommended that it examine National Assessment 
materials and methodologies in detail to determine their possible use 
within urban school systems. This type of cooperation, examination 
and action will make National Assessment data more relevant to urban 



needs and increase its potential for use in decision making within 
local, urban school districts. 

For a new program Model Utilization and Applications has 
generated an impressive amount of interest and a large number of 
intermediate products. At least 14 states and 5 local districts have 
actually adapted some of the NAEP materials, procedures, or technologies 
in conducting their own assessments. The high level of early interest 
and experimentation has occurred because NAEP's Model Utilization and 
Applications is one source that might be used to solve some difficult 
educational problems. The political and lay publics have been 
applying more and more pressure on educators to justify educational 
expenditures in terms of educational outcoTr.cs. IJhen this pressure 
comes from the lay community or from a state legislature, it is usually 
called accountability; and when it comes from the United States 
Congress 5 it is usually called Title "x" evaluation. 

Many of the educators who accept this pressure as fair or who 
know it will soon necessitate justification of expenditures in terms 
of educational outcomes do not believe traditional testing instruments 
can provide a fair judgment. Furthermore, many educators vjere seeking 
a supplement to standardized tests long before accountability pressures 
began. The National Assessment model fills a void left by standardized 
tests which measure individual student differences and establish, 
reliable average scores for grades or schools based upon the 
perforaance of average students on cognitive test items. National 
Assessment provides direct information about the performances of 
various population groups in terms of desirable knowledges, skills, 
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understandings and attitudes. This type of information can provide 

educators with data about valuable educational achievements within 

their agencies and with data about weaknesses which demand a greater 

allocation of resources. 

If one views the data about the educational attainments of young 

Americans as the product of National Assessment, then the great 
advancement in the technologies and methodologies of objectives-referenced 

assessment and the actual objectives and exercises published by NAEP 
must be viewed as two very valuable by-products. National Assessment 
has developed procedures for producing concensus objectives and 
selecting pools. of exercises by achieving agreement among scholars, 
educators and laymen; it has developed- criteria for producing and 
scoring objectives-referenced exercises to measure very diverse 
objectives; it has developed procedures for administering objectives- 
referenced exercises to minimize interference from extraneous factors 
and maximize valid results; and it has developed a sampling and 
analysis plan to yield direct, comparative educational data about 
population groups , 

Therefore, National Assessment was certainly one logical place 
for states and local districts to contact about producing data that 
could be related to group performance on specific objectives. I^hile 
National Assessment and the Education Commission of the States firmly 

resisted any pressures to gather and report data pertaining to 
performance within particular state or local education agencies, they 

did establish, through Model Utilization and Applications, a program 

to make NAEP materials, procedures and technologies available to 



interested state and local education agencies. Because of the state 
thrust of ECS and a- limited amount of resources, consultation and 
technical assistance are provided most frequently to states; however, 
they are available to local districts which are an integral part of a 
state assessment program or which are involved in a unique demonstration 
of the adaptability of the NAEP model at the local level. 

National Assessment consultation and technical assistance takes 
place in annual workshops or with staff from individual education 
agencies • The National Assessment model and existing state and local 
adaptations are described and the luany problems of designing and 
implementing an assessment are discussed. If an educational agency 
decid/as to use National Assessment materials or adapt some of the 
procedures or technologies, technical assistance is provided by 
appropriate members of the NAEP staff. 

State and local adaptations have varied considerably depending 

upon the goals of the agency. For instance, Connecticut and Maine 

conducted assessments relying exclusively upon National Assessment 

exercises and using sampling, administration, scoring and analysis 

procedures that compared state results with NAEP's national and 

regional resuxts. But Connecticut related NAEP exercises and 

objectives to existing state objeictives while Maine did not. And Maine 
related student performance to many non-school factors not investigated 

by NAEP or Connecticut. Massachusetts attempted to avoid state 
comparisons with national data. National Assessment objectives were 
prioritized by a sample of teachers, and exercises (including f;ome not 
developed by NAEP) measuring the priority objectives were administered 



to a sample of grade-level students using traditional administrative 
methods. The data was looked at in tenns of the level of student 
performance as related to the expectations of Massachusetts educators. 

At the local level both Lincoln, Nebraska and Montgomery County, 
Maryland also plan to conduct assessments relying exclusively upon 
National Assessment exercises and using sampling, administration, 
scoring and analysis procedures which will allow local results to be 
compared with national and regional results. However, San Bernardino, 
California developed a model similar to that of National Assessment 
and used local teachers tG develop objectives and exercises. San 
Bernardino used the services of Model Utilization and Applications as 
a source for technological and methodological consultation and 
borrowed from NAEP objectives and exercises only as they applied to 
the locally developed model. 

Specific program changes based upon these and other early efforts 
are not available; however, Maine and Connecticut have begun to look 
more closely at certain aspects of their curriculums based upon results, 
and Massachusetts has achieved an ir.crea''ved interest in mastery level 
testing. Interest in various adaptations has continued to increase. 
Many states and local districts which have conducted one assessment are 
planning to conduct another, usually in a different subject area and with 
increased technical sophistication. Some states and local districts 
are planning a first attempt; some states are designing_;loug run, 
cyclical state assessments which will require high levels of state 
funding, e.g. Minnesota and Illinois. Also, in some states which have 
conducted state assessments, local districts have inquired about 
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receiving data on local performance levels as compared to state 
performance levels. 

The edugatiom. ..uuptations of NAEP's objecti- 

referenced model at state and local levels depends upon many things. 
The early, experimental efforts must reveal some actual, and much 
potential utility for revealing educational strengths and weaknesses 
to educators and laymen. Within a reasonable amount of time, some 
states and localities will have to use these early efforts to design 
and operationalize sophisticated, well-funded assessments capable of 
providing data for the successful allocation of resources at the 
program level. This will require a relatively heavy financial 
commitment from the state or local agency. One reason National 
Assessment has been a leader in the advancement of objectives-referenced 
technologies, methodologies and materials is because objectives- 
referenced assessment is expensive. The investment made through 
National Assessment can cause state and local developmental and 
operational efforts to be a lot easier and much less expensive than 
they would have been; however. National Assessment cannot provide 
various states and localities with packaged, inexpensive assessments 
for allocating resources according to unique needs. Educators who 
believe objectives-referenced assessment would be beneficial at the 
state or local level and educators who are pressured to justify 
expenditures according to educational outcomes must convince those who 
control their budgets to give them the funds necessary for designing 
and implementing an assessment which will provide data that can be 
ERJC used for fair program evaluation within their agencies. 



The program for Model Utilization and Applications is designed to 

contribute to the development of more sophisticated assessments. As 

state and local education agencies secure adequate assessment funding 

and produce data which raeets local needs National Assessment \^all 

continue to share its products and give consultation and technical 

assistance based upon the latest methodologies and technologies. It 

will also provide a forum for those involved in state and local 

assessments in order to advance assessment technology, and it will 
disseminate both the products of local assessments and the latest 

technological and methodological developments. 

National Assessment has not had a major impact on American 
education to date, but it has had some impact. If significant 
educational implications can be found in the great mass of data already 
produced and if state and local education agencies can successfully 
adapt parts of the NAEP model to their needs, the project can have 
considerable short run effects on state and local education. The 
progress of the Department of Utilization and Applications to this 
point In time does not guarantee this, but its active programs and 
early accomplishments do make it very possible. 

Before the question of whether the National Assessment of 

Educational Progress is having enough short run impact on state and 
local education is answered, the question should be put in perspective. 

It is certainly important, but its importance must not be over- 
emphasized. Two other questions are more important. If National 
Assessment is not having enough impact on state and local education, 
is the successful provision of continuous comprehensive data on the 



educational attainments of young i\Tnericans worth the money spent on 
the project? If National Assessment has significant impact on state 
and local assessment, will it be possible to continue to provide 
comprehensive dr n the educational attainments of young Americans 
while workinj.; undc ae pressures of increasing that impact each 
year and dealing with the inherent controversies? Unless policy 
makers alter National Assessment's basic goals the project should 
be evaluated primarily according to its success in meeting those 
goals. Otherwise, a project mth great long run value to American 
education might be discontinued before it has had a chance to make 
a return on its investment, or it might be altered to maximize its 
return on secondary goals rather than primary ones. 
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